Predictions and errors are distinctly represented across V1 layers.

Popular accounts of mind and brain propose that the brain continuously forms predictions about future sensory inputs and combines predictions with inputs to determine what we perceive.1,2,3,4,5,6 Under "predictive processing" schemes, such integration is supported by the hierarchical organization of the cortex, whereby feedback connections communicate predictions from higher-level deep layers to agranular (superficial and deep) lower-level layers.7,8,9,10 Predictions are compared with input to compute the "prediction error," which is transmitted up the hierarchy from superficial layers of lower cortical regions to the middle layers of higher areas, to update higher-level predictions until errors are reconciled.11,12,13,14,15 In the primary visual cortex (V1), predictions have thereby been proposed to influence representations in deep layers while error signals may be computed in superficial layers. Despite the framework's popularity, there is little evidence for these functional distinctions because, to our knowledge, unexpected sensory events have not previously been presented in human laminar paradigms to contrast against expected events. To this end, this 7T fMRI study contrasted V1 responses to expected (75% likely) and unexpected (25%) Gabor orientations. Multivariate decoding analyses revealed an interaction between expectation and layer, such that expected events could be decoded with comparable accuracy across layers, while unexpected events could only be decoded in superficial laminae. Although these results are in line with these accounts that have been popular for decades, such distinctions have not previously been demonstrated in humans. We discuss how both prediction and error processes may operate together to shape our unitary perceptual experiences.

Expectation Cues and False Percepts Generate Stimulus-Specific Activity in Distinct Layers of the Early Visual Cortex.

Perception has been proposed to result from the integration of feedforward sensory signals with internally generated feedback signals. Feedback signals are believed to play an important role in driving false percepts, that is, seeing things that are not actually there. Feedforward and feedback influences on perception can be studied using layer-specific fMRI, which we used here to interrogate neural activity underlying high-confidence false percepts while healthy human participants (N = 25, male and female) performed a perceptual orientation discrimination task. Auditory cues implicitly signaled the most likely upcoming orientation (referred to here as expectations). These expectations induced orientation-specific templates in the deep and superficial layers of V2, without affecting perception. In contrast, the orientation of falsely perceived stimuli with high confidence was reflected in the middle input layers of V2, suggesting a feedforward signal contributing to false percepts. The prevalence of high-confidence false percepts was related to everyday hallucination severity in a separate online sample (N = 100), suggesting a possible link with abnormal perceptual experiences. These results reveal a potential feedforward mechanism underlying false percepts, reflected by spontaneous stimulus-like activity in the input layers of the visual cortex, independent of top-down signals reflecting cued orientations.SIGNIFICANCE STATEMENT False percepts have been suggested to arise through excessive feedback signals. However, feedforward contributions to false percepts have remained largely understudied. Laminar fMRI has been shown to be useful in distinguishing feedforward from feedback activity as it allows the imaging of different cortical layers. In the present study we demonstrate that although cued orientations are encoded in the feedback layers of the visual cortex, the content of the false percepts are encoded in the feedforward layers and did not rely on these cued orientations. This shows that false percepts can in principle emerge from random feedforward signals in the visual cortex, with possible implications for disorders hallmarked by hallucinations like schizophrenia and Parkinson's disease.

Stubborn Predictions in Primary Visual Cortex.

Perceivers can use past experiences to make sense of ambiguous sensory signals. However, this may be inappropriate when the world changes and past experiences no longer predict what the future holds. Optimal learning models propose that observers decide whether to stick with or update their predictions by tracking the uncertainty or "precision" of their expectations. However, contrasting theories of prediction have argued that we are prone to misestimate uncertainty-leading to stubborn predictions that are difficult to dislodge. To compare these possibilities, we had participants learn novel perceptual predictions before using fMRI to record visual brain activity when predictive contingencies were disrupted-meaning that previously "expected" events became objectively improbable. Multivariate pattern analyses revealed that expected events continued to be decoded with greater fidelity from primary visual cortex, despite marked changes in the statistical structure of the environment, which rendered these expectations no longer valid. These results suggest that our perceptual systems do indeed form stubborn predictions even from short periods of learning-and more generally suggest that top-down expectations have the potential to help or hinder perceptual inference in bounded minds like ours.

Hippocampal representations switch from errors to predictions during acquisition of predictive associations.

We constantly exploit the statistical regularities in our environment to help guide our perception. The hippocampus has been suggested to play a pivotal role in both learning environmental statistics, as well as exploiting them to generate perceptual predictions. However, it is unclear how the hippocampus balances encoding new predictive associations with the retrieval of existing ones. Here, we present the results of two high resolution human fMRI studies (N = 24 for both experiments) directly investigating this. Participants were exposed to auditory cues that predicted the identity of an upcoming visual shape (with 75% validity). Using multivoxel decoding analysis, we find that the hippocampus initially preferentially represents unexpected shapes (i.e., those that violate the cue regularities), but later switches to representing the cue-predicted shape regardless of which was actually presented. These findings demonstrate that the hippocampus is involved both acquiring and exploiting predictive associations, and is dominated by either errors or predictions depending on whether learning is ongoing or complete.

Imagery adds stimulus-specific sensory evidence to perceptual detection.

Internally generated imagery and externally triggered perception rely on overlapping sensory processes. This overlap poses a challenge for perceptual reality monitoring: determining whether sensory signals reflect reality or imagination. In this study, we used psychophysics to investigate how imagery and perception interact to determine visual experience. Participants were instructed to detect oriented gratings that gradually appeared in noise while simultaneously either imagining the same grating, a grating perpendicular to the to-be-detected grating, or nothing. We found that, compared to both incongruent imagery and no imagery, congruent imagery caused a leftward shift of the psychometric function relating stimulus contrast to perceptual threshold. We discuss how this effect can best be explained by a model in which imagery adds sensory signal to the perceptual input, thereby increasing the visibility of perceived stimuli. These results suggest that, in contrast to changes in sensory signals caused by self-generated movement, the brain does not discount the influence of self-generated sensory signals on perception.

Perceptual reality monitoring: Neural mechanisms dissociating imagination from reality.

There is increasing evidence that imagination relies on similar neural mechanisms as externally triggered perception. This overlap presents a challenge for perceptual reality monitoring: deciding what is real and what is imagined. Here, we explore how perceptual reality monitoring might be implemented in the brain. We first describe sensory and cognitive factors that could dissociate imagery and perception and conclude that no single factor unambiguously signals whether an experience is internally or externally generated. We suggest that reality monitoring is implemented by higher-level cortical circuits that evaluate first-order sensory and cognitive factors to determine the source of sensory signals. According to this interpretation, perceptual reality monitoring shares core computations with metacognition. This multi-level architecture might explain several types of source confusion as well as dissociations between simply knowing whether something is real and actually experiencing it as real. We discuss avenues for future research to further our understanding of perceptual reality monitoring, an endeavour that has important implications for our understanding of clinical symptoms as well as general cognitive function.

Mistaking imagination for reality: Congruent mental imagery leads to more liberal perceptual detection.

Visual experiences can be triggered externally, by signals coming from the outside world during perception; or internally, by signals from memory during mental imagery. Imagery and perception activate similar neural codes in sensory areas, suggesting that they might sometimes be confused. In the current study, we investigated whether imagery influences perception by instructing participants to imagine gratings while externally detecting these same gratings at threshold. In a series of three experiments, we showed that imagery led to a more liberal criterion for reporting stimulus presence, and that this effect was both independent of expectation and stimulus-specific. Furthermore, participants with more vivid imagery were generally more likely to report the presence of external stimuli, independent of condition. The results can be explained as either a low-level sensory or a high-level decision-making effect. We discuss that the most likely explanation is that during imagery, internally generated sensory signals are sometimes confused for perception and suggest how the underlying mechanisms can be further characterized in future research. Our findings show that imagery and perception interact and emphasize that internally and externally generated signals are combined in complex ways to determine conscious perception.

Prior expectations evoke stimulus-specific activity in the deep layers of the primary visual cortex.

The way we perceive the world is strongly influenced by our expectations. In line with this, much recent research has revealed that prior expectations strongly modulate sensory processing. However, the neural circuitry through which the brain integrates external sensory inputs with internal expectation signals remains unknown. In order to understand the computational architecture of the cortex, we need to investigate the way these signals flow through the cortical layers. This is crucial because the different cortical layers have distinct intra- and interregional connectivity patterns, and therefore determining which layers are involved in a cortical computation can inform us on the sources and targets of these signals. Here, we used ultra-high field (7T) functional magnetic resonance imaging (fMRI) to reveal that prior expectations evoke stimulus-specific activity selectively in the deep layers of the primary visual cortex (V1). These findings are in line with predictive processing theories proposing that neurons in the deep cortical layers represent perceptual hypotheses and thereby shed light on the computational architecture of cortex.

Management of Respiratory Motion Artefacts in 18F-fluorodeoxyglucose Positron Emission Tomography using an Amplitude-Based Optimal Respiratory Gating Algorithm.

Positron emission tomography (PET) combined with X-ray computed tomography (CT) is an important molecular imaging platform that is required for accurate diagnosis and clinical staging of a variety of diseases. The advantage of PET imaging is the ability to visualize and quantify a myriad of biological processes in vivo with high sensitivity and accuracy. However, there are multiple factors that determine image quality and quantitative accuracy of PET images. One of the foremost factors influencing image quality in PET imaging of the thorax and upper abdomen is respiratory motion, resulting in respiration-induced motion blurring of anatomical structures. Correction of these artefacts is required for providing optimal image quality and quantitative accuracy of PET images. Several respiratory gating techniques have been developed, typically relying on acquisition of a respiratory signal simultaneously with PET data. Based on the respiratory signal acquired, PET data is selected for reconstruction of a motion-free image. Although these methods have been shown to effectively remove respiratory motion artefacts from PET images, the performance is dependent on the quality of the respiratory signal being acquired. In this study, the use of an amplitude-based optimal respiratory gating (ORG) algorithm is discussed. In contrast to many other respiratory gating algorithms, ORG permits the user to have control over image quality versus the amount of rejected motion in the reconstructed PET images. This is achieved by calculating an optimal amplitude range based on the acquired surrogate signal and a user-specified duty cycle (the percentage of PET data used for image reconstruction). The optimal amplitude range is defined as the smallest amplitude range still containing the amount of PET data required for image reconstruction. It was shown that ORG results in effective removal of respiration-induced image blurring in PET imaging of the thorax and upper abdomen, resulting in improved image quality and quantitative accuracy.

Prior Expectations of Motion Direction Modulate Early Sensory Processing.

Perception is a process of inference, integrating sensory inputs with prior expectations. However, little is known regarding the temporal dynamics of this integration. It has been proposed that expectation plays a role early in the perceptual process, biasing sensory processing. Alternatively, others suggest that expectations are integrated only at later, postperceptual decision-making stages. The current study aimed to dissociate between these hypotheses. We exposed human participants (male and female) to auditory cues predicting the likely direction of upcoming moving dot patterns, while recording neural activity using magnetoencephalography (MEG). Participants' reports of the moving dot directions were biased toward the direction predicted by the cues. To investigate when expectations affected sensory representations, we used inverted encoding models to decode the direction represented in early sensory signals. Strikingly, the cues modulated the direction represented in the MEG signal as early as 150 ms after visual stimulus onset. While this may not reflect a modulation of the initial feedforward sweep, it does reveal a modulation of early sensory representations. Exploratory analyses showed that the neural modulation was related to perceptual expectation effects: participants with a stronger perceptual bias toward the predicted direction also revealed a stronger reflection of the predicted direction in the MEG signal. For participants with this perceptual bias, a correlation between decoded and perceived direction already emerged before visual stimulus onset, suggesting that the prestimulus state of the visual cortex influences sensory processing. Together, these results suggest that expectations play an integral role in the neural computations underlying perception.SIGNIFICANCE STATEMENT Perception can be thought of as an inferential process in which our brains integrate sensory inputs with prior expectations to make sense of the world. This study investigated whether this integration occurs early or late in the process of perception. We exposed human participants to auditory cues that predicted the likely direction of visual moving dots, while recording neural activity with millisecond resolution using magnetoencephalography. Participants' perceptual reports of the direction of the moving dots were biased toward the predicted direction. Additionally, the predicted direction modulated the neural representation of the moving dots just 150 ms after they appeared. This suggests that prior expectations affected sensory processing at early stages, playing an integral role in the perceptual process.

Content-based Dissociation of Hippocampal Involvement in Prediction.

Recent work suggests that a key function of the hippocampus is to predict the future. This is thought to depend on its ability to bind inputs over time and space and to retrieve upcoming or missing inputs based on partial cues. In line with this, previous research has revealed prediction-related signals in the hippocampus for complex visual objects, such as fractals and abstract shapes. Implicit in such accounts is that these computations in the hippocampus reflect domain-general processes that apply across different types and modalities of stimuli. An alternative is that the hippocampus plays a more domain-specific role in predictive processing, with the type of stimuli being predicted determining its involvement. To investigate this, we compared hippocampal responses to auditory cues predicting abstract shapes (Experiment 1) versus oriented gratings (Experiment 2). We measured brain activity in male and female human participants using high-resolution fMRI, in combination with inverted encoding models to reconstruct shape and orientation information. Our results revealed that expectations about shape and orientation evoked distinct representations in the hippocampus. For complex shapes, the hippocampus represented which shape was expected, potentially serving as a source of top-down predictions. In contrast, for simple gratings, the hippocampus represented only unexpected orientations, more reminiscent of a prediction error. We discuss several potential explanations for this content-based dissociation in hippocampal function, concluding that the computational role of the hippocampus in predictive processing may depend on the nature and complexity of stimuli.

Spatiotemporal dynamics of brightness coding in human visual cortex revealed by the temporal context effect.

Human visual perception is modulated by both temporal and spatial contexts. One type of modulation is apparent in the temporal context effect (TCE): In the presence of a constant luminance patch (a long flash), the perceived brightness of a short flash increases monotonically with onset asynchrony. The aim of the current study was to delineate the neural correlates of this illusory effect, particularly focusing on its dynamic neural representation among visual cortical areas. We reconstructed sources of magnetoencephalographic (MEG) data recorded from observers (6 male and 9 female human adults) experiencing the TCE. Together with retinotopic mapping, signals from different occipital lobe areas were extracted to investigate whether different visual areas have differential representation of the onset vs. offset synchronized short flashes. From the data, TCE related responses were observed in LO and V4 in the time window of 200-250 m s, while neuronal responses to physical luminances were observed in the early time window at around 100 m s across early visual cortex, such as V1 and V2, also in V4 and VO. Based on these findings, we suggest that two distinct processes might be involved in brightness coding: one bottom-up process which is stimulus energy driven and responds fast, and another process which may be broadly characterized as top-down or lateral, is context driven, and responds slower. For both processes, we found that V4 might play a critical role in dynamically integrating luminances into brightness perception, a finding that is consistent with the view of V4 as a bottom-up and top-down integration complex.

The Perceptual Prediction Paradox.

From the noisy information bombarding our senses, our brains must construct percepts that are veridical - reflecting the true state of the world - and informative - conveying what we did not already know. Influential theories suggest that both challenges are met through mechanisms that use expectations about the likely state of the world to shape perception. However, current models explaining how expectations render perception either veridical or informative are mutually incompatible. While the former propose that perceptual experiences are dominated by events we expect, the latter propose that perception of expected events is suppressed. To solve this paradox we propose a two-process model in which probabilistic knowledge initially biases perception towards what is likely and subsequently upweights events that are particularly surprising.

No evidence for altered up- and downregulation of brain activity in visual cortex during illusory shape perception in autism.

Autism spectrum disorder (ASD) may be marked by an altered balance between sensory input and prior expectations. Because many illusions rely on integrating sensory input with prior information such as spatial context, individuals with ASD may therefore be less susceptible to visual illusions than typically developing (TD) individuals. Yet empirical evidence on the matter is rather divergent, varying depending on the type of illusion, study procedure, and population. Visual illusions lead to neural activity alterations in the visual system. In the so-called Kanizsa illusion, these are likely caused by top-down feedback to V1. Here we tested the hypothesis that a reduced susceptibility to illusions in ASD would manifest as diminished modulation of V1 activity by illusions, using functional magnetic resonance imaging (fMRI). We presented 22 adolescents with ASD and 22 age-, gender-, and intelligence-matched TD controls with displays that consisted of three circular inducers. These either formed an illusory triangle (Kanizsa illusion) or not. We identified regions in primary visual cortex (V1) that corresponded to (the visual field locations of) the illusory triangle and its inducers, and recorded their visual response. Previous research in healthy volunteers has shown a specific pattern of up- and down-regulation in regions of V1 that process the shape and inducers, respectively. Here, we replicated this pattern of up- and downregulation in V1, in both the TD and ASD groups, with no differences between groups. This suggests that illusory shape processing in primary visual cortex is equally present in ASD, suggesting unimpaired processing of spatial context.

Laminar Organization of Working Memory Signals in Human Visual Cortex.

The human primary visual cortex (V1) is not only activated by incoming visual information but is also engaged by top-down cognitive processes, such as visual working memory, even in the absence of visual input [1-3]. This feedback may be critical to our ability to visualize specific visual features, as higher-order regions lack the selectivity to represent such information [4]. Clearly, such internally generated signals do not trigger genuine perception of the remembered stimulus, meaning they must be organized in a manner that is different to bottom-up-driven signals. Internally generated signals may be kept separate from incoming sensory data by virtue of the laminar organization of inter-area cortical connections. Namely, bottom-up driving connections target layer 4, located in the middle of the cortical column, and feedback connections target deep and superficial layers and avoid layer 4 [5-7]. Using lamina-resolved fMRI, we simultaneously measured the activity in three early visual cortical areas (V1-V3) that are recruited to represent stimulus information during visual working memory [8]. We observed item-specific working memory signals in early visual cortex. In V1, this item-specific activity was selectively present at deep and superficial cortical depths, avoiding the middle layers, and working-memory-related activity was present at all depths in V2 and V3. These results show for the first time the laminar organization of internally generated signals during visual working memory in the human visual system and provide new insights into how bottom-up and top-down signals in visual cortex are deployed.

Eye Movement-Related Confounds in Neural Decoding of Visual Working Memory Representations.

A relatively new analysis technique, known as neural decoding or multivariate pattern analysis (MVPA), has become increasingly popular for cognitive neuroimaging studies over recent years. These techniques promise to uncover the representational contents of neural signals, as well as the underlying code and the dynamic profile thereof. A field in which these techniques have led to novel insights in particular is that of visual working memory (VWM). In the present study, we subjected human volunteers to a combined VWM/imagery task while recording their neural signals using magnetoencephalography (MEG). We applied multivariate decoding analyses to uncover the temporal profile underlying the neural representations of the memorized item. Analysis of gaze position however revealed that our results were contaminated by systematic eye movements, suggesting that the MEG decoding results from our originally planned analyses were confounded. In addition to the eye movement analyses, we also present the original analyses to highlight how these might have readily led to invalid conclusions. Finally, we demonstrate a potential remedy, whereby we train the decoders on a functional localizer that was specifically designed to target bottom-up sensory signals and as such avoids eye movements. We conclude by arguing for more awareness of the potentially pervasive and ubiquitous effects of eye movement-related confounds.

How Do Expectations Shape Perception?

Perception and perceptual decision-making are strongly facilitated by prior knowledge about the probabilistic structure of the world. While the computational benefits of using prior expectation in perception are clear, there are myriad ways in which this computation can be realized. We review here recent advances in our understanding of the neural sources and targets of expectations in perception. Furthermore, we discuss Bayesian theories of perception that prescribe how an agent should integrate prior knowledge and sensory information, and investigate how current and future empirical data can inform and constrain computational frameworks that implement such probabilistic integration in perception.

Associative Prediction of Visual Shape in the Hippocampus.

Perception can be cast as a process of inference, in which bottom-up signals are combined with top-down predictions in sensory systems. In line with this, neural activity in sensory cortex is strongly modulated by prior expectations. Such top-down predictions often arise from cross-modal associations, such as when a sound (e.g., bell or bark) leads to an expectation of the visual appearance of the corresponding object (e.g., bicycle or dog). We hypothesized that the hippocampus, which rapidly learns arbitrary relationships between stimuli over space and time, may be involved in forming such associative predictions. We exposed male and female human participants to auditory cues predicting visual shapes, while measuring high-resolution fMRI signals in visual cortex and the hippocampus. Using multivariate reconstruction methods, we discovered a dissociation between these regions: representations in visual cortex were dominated by whichever shape was presented, whereas representations in the hippocampus reflected only which shape was predicted by the cue. The strength of hippocampal predictions correlated across participants with the amount of expectation-related facilitation in visual cortex. These findings help bridge the gap between memory and sensory systems in the human brain.SIGNIFICANCE STATEMENT The way we perceive the world is to a great extent determined by our prior knowledge. Despite this intimate link between perception and memory, these two aspects of cognition have mostly been studied in isolation. Here we investigate their interaction by asking how memory systems that encode and retrieve associations can inform perception. We find that upon hearing a familiar auditory cue, the hippocampus represents visual information that had previously co-occurred with the cue, even when this expectation differs from what is currently visible. Furthermore, the strength of this hippocampal expectation correlates with facilitation of perceptual processing in visual cortex. These findings help bridge the gap between memory and sensory systems in the human brain.

Stimulus Familiarity and Expectation Jointly Modulate Neural Activity in the Visual Ventral Stream.

Prior knowledge about the visual world can change how a visual stimulus is processed. Two forms of prior knowledge are often distinguished: stimulus familiarity (i.e., whether a stimulus has been seen before) and stimulus expectation (i.e., whether a stimulus is expected to occur, based on the context). Neurophysiological studies in monkeys have shown suppression of spiking activity both for expected and for familiar items in object-selective inferotemporal cortex. It is an open question, however, if and how these types of knowledge interact in their modulatory effects on the sensory response. To address this issue and to examine whether previous findings generalize to noninvasively measured neural activity in humans, we separately manipulated stimulus familiarity and expectation while noninvasively recording human brain activity using magnetoencephalography. We observed independent suppression of neural activity by familiarity and expectation, specifically in the lateral occipital complex, the putative human homologue of monkey inferotemporal cortex. Familiarity also led to sharpened response dynamics, which was predominantly observed in early visual cortex. Together, these results show that distinct types of sensory knowledge jointly determine the amount of neural resources dedicated to object processing in the visual ventral stream.